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Abstract 

Synaptic efficacy between neurons is known to change within a short time scaie 
dynamically. Ncurophysiological experiments show that high-frequency presy- 
naptic inputs decrease synaptic efficacy between neurons. This phenomenon is 
called synaptic depression, a short term synaptic plasticity. Many researchers 
have investigated how the synaptic depression affects the memory storage capac- 
ity. However, the noise has not been taken into consideration in their analysis. 
By introducing "temperature", which controls the level of the noise, into an 
update rule of neurons, we investigate the effects of synaptic depression on the 
memory storage capacity in the presence of the noise. We analytically compute 
the storage capacity by using a statistical mechanics technique called Self Con- 
sistent Signal to Noise Analysis (SCSNA). We find that the synaptic depression 
decreases the storage capacity in the case of finite temperature in contrast to the 
case of the low temperature limit, where the storage capacity does not change. 
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1 Introduction 



Synaptic efficacy between neurons is known to change within a short time scaie 
dynamically. Ncurophysiological experiments show that high-frequency presy- 
naptic inputs decrease synaptic efficacy between neurons. This phenomenon 
is called synaptic depression, a short term synaptic plasticity. pQ [2] [3] Many 
researchers have investigated how the synaptic depression affects the retrieval 
property of the associative memory model, jl] [5] The associative memory model 
is a typical artificial neural network model that has discretely distributed fixed- 
point attractors as stored patterns. [5] Otsubo et al. showed that synaptic 
depression destabilizes a steady state, and then the network oscillates among 
attractors analytically. [5] Although their research focused on network stored 
finite memory patterns, we should focus on the infinite version in order to dis- 
cuss the general retrieval property of network. In the case that synaptic strength 
is static, the associative memory model has the maximum number of memory 
patterns that the network is able to retrieve. The number of such patterns 
per neuron is called storage capacity. This is important quantity for discussing 
memory performance [7]. It is interesting problem how short term synaptic plas- 
ticity affects the stability of network with infinite attractors. Therefore, we 
focus on the storage capacity in order to discuss the retrieval property of the 
network with synaptic depression in the case of infinite memory patterns. There 
have been previous investigations into how the synaptic depression influences 
the storage capacity. [8 9 [10 Matsumoto et al. showed that the synaptic de- 
pression does not affect the storage capacity in the low temperature limit. [5] 
However, the noise has not been taken into consideration in their analysis. The 
storage capacity is known to depend on "temperature" , which controls the level 
of the noise. [TT] 

In this paper, we investigate the effects of synaptic depression on the memory 
storage capacity in the presence of temperature. We analytically compute the 
storage capacity by using a statistical mechanics technique called Self Consistent 
Signal to Noise Analysis (SCSNA), which is an extended method of signal to 
noise analysis [12]. This allows us to discuss the retrieval property in the network 
with synaptic depression more generally. 

This paper consists of five sections. Section 2 describes the model used in 
this paper. Section 3 introduces the analysis by SCSNA in order to obtain order 
parameter equations. Section 4 investigates the steady state of network with 
synaptic depression by theory and computer simulation. Section 5 summarizes 
the results obtained in this paper discusses them. 



We consider a recurrent neural network model with N fully connected analogue 
neurons. The state of i-th neuron mi(t), which has a continuous value between 
and 1, i.e., [0, 1], and an internal potential hi(t) at time t are updated by the 
following rule [IB]: 
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where Jij (t) represents the synaptic weight between i-th neuron and j-th neuron 
at time t, and F(-) is an output function. Thus, the state of network at time 
t is characterized by m(t) = (mi(t),--- , mjv(i))- The synaptic weight Jy(t) 
incorporating synaptic depression can be described by 

Jij(t) = JijXj(t), (3) 

Xj (t+1) = Xj (t) + 1 ~ * 3 '® - U^Xj^mjit), (4) 

where Xj(t) is determined by a phenomenological model of synapses, [21 131 E] and 
takes < Xj{t) < 1, where Xj(t) = 1 corresponds to the case without synaptic 
depression. The depression variable Xj(t + 1) from the presynaptic neuron is 
decreased by a certain fraction UsEXj(t) and recovers with time constant t. 
The fixed synaptic weight Jy is, according to the Hebbian rule, set to 

aN 

where aN corresponds to the number of memory patterns, and a is called 
the loading rate, which is the ratio between the number of embedded memory 
patterns and neurons. Each element of the memory pattern £ M = (£5*, • • • is 
generated randomly and independently by 

Prob^ = ±1] = \. (6) 

3 Analysis of Steady State by SCSNA 

In this section, we derive macroscopic parameters in steady state in the case of 
a ~ O(l) by using SCSNA [12], which is an extension of a naive signal to noise 
(S/N) analysis. 

Considering the steady states, TOi(oo) = rrii, hi(oo) = hi and Xi(oo) = x- t , 
we obtain the following equations from eqs. (JIJ and ((3]), 

mi = F(ht), (7) 
hi = J.j.r.m,. (8) 
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1 + 7TOj 



(7 = U se t), (9) 



where 7 indicates the level of synaptic depression in the steady state. Here, we 
use the following transformation, 

r,- = i,m; = — , (10) 

3 3 3 1 + 7771/ v ' 

and then we obtain the following relationship concerning Tj and internal poten- 
tial hj for cq. © and ©, 

Fjhj) 



hj = ^JijTj. (12) 
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We introduce the following order parameter, 



< - as) 

i=l 

where 7r ? f corresponds to overlap between r and £ M . If the state of network 
retrieve a memory pattern the above order parameter is exactly 1 at N — > oo, 
i.e., 7r£f = 1. By using rj in place of m_j, the effect of synaptic depression is 
incorporated in the function G(-). Thus, we can apply SCSNA to our model 
under the transformation (flT)|) . The basis of the SCSNA is in the systematic 
splitting of the internal potential into a signal and a cross-talk noise. Moreover, 
the cross-talk noise part consists of two elements. One is an effective self- 
coupling term that comes from statistical correlations caused by the recurrent 
connections, and the other obeys Gaussian distribution. By this scheme, the 
order parameter equations are obtained as follows: 

y = G^|^ + « + ryV (2-^(0,1)). (14) 

7r r = 2(l+ 7 ) / Dz{{&)). (15) 

(16) 

(17) 

(18) 
(19) 

The variable Y represents the effective output for r,; and the stochastic variable 
£, obeying cq. ([5]), corresponds to a retrieving pattern component £ . The 
order parameters ir r correspond to overlap 7T* . Note that the bracket ((■)) mean 
the expectation for stochastic variable £ and Dz = l/v / 27rcxp(— z 2 /2). The 
notation J¥(Q, 1) represents Gaussian distribution with mean 1 variance. Each 
argument of the function G(-) consists of three terms. The first term, 2(1+7) ' 
comes from a signal component, the second term is assumed to be Gaussian 
distribution with mean a 1 variance, and the third term means systematic bias 
of crosstalk noise. The detailed derivation of the above equations is given in the 
appendix. 

We should also consider the overlap between the state of network, m, and 
fi — th memory pattern, in order to describe the macroscopic state of network 
at a steady state, 

i N 

<=-E^(2m,-l). (20) 
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By using rj, the overlap can be described as 

+ 7)- 
I-7Y 
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Figure 1: The form of and G(/i) in the case of the sigmoidal output 

function (22) with j3 = 2.0 and 7 = 0.5 as an example. Dashed line in left figure 
represents value of G(h) in limit of large h for eq. (fTTj). i.e., 1/(1 + 7). 



where ir m = tt^. 

In this paper we consider the output function as sigmoidal, 

F(h) = -(l + tanh/S/i). ( 22 ) 

Here, /3 = ^ represents the inverse temperature, and the neuron state, rrii, 
takes a binary value 1 or in /? — > 00. The previous work estimated the 
storage capacity under the following relationship at the low temperature limit 
of 00 [9]: 

TOo TO," 

a; 7 -TO, = 3 - — = — - J -, 23) 

13 1 + jrrij 1 + 7 

F(ft) = 9(/i), (24) 
where 0(/i) is a step function: 

Therefore, we obtain the relationship, to^ = 0(^3 Jij m jl (1+7)) = ®{J2j^i Jij m j) 
by using eqs. (10) and (12). This equation is same as one without synaptic de- 
pression. That means the storage capacity does not change in (3 — > 00. In the 
case of finite j3, the convergent point and the slope of G(h) is different from 
those of F(h) (Fig. 1). For these reasons, the storage capacity is expected to 
change at finite temperature. 

3.1 Results 

In this section, we investigate the storage capacity in a steady state for finite 
temperature by using theory and computer simulation numerically. We use the 
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sigmoid function, F(-) = (1 + tanh/3(-))/2, for the output function. Using sig- 
moid function make us to find several solution candidates in ea. (|14|) . so that 
we should introduce the Maxwell rule for choosing a unique solution|12|. Com- 
puter simulations are performed with N = 5000 and random patterns 
following cq. ([6]) in 11 trials. The initial state in computer simulations is set to 
mi{0) = (£ + l)/2 and a;<(0) = 1, i.e., n m = 1,7^ = 1. 

Figure [5] (a) and (b) show how the overlap n m defined by eq. (|2T)|) depends 
on the loading rate a at T = 0.1. Figure [2] (a) and (b) are the cases without 
and with synaptic depression, i.e., 7 = 0.5, r = 2.0. The solid lines show 
the solutions obtained by solving eqs. (|14l) - (fT9]) and (|2"Tj) numerically while 
error bars indicate medians with minimum and maximum values obtained by 
computer simulation in 11 trials at N = 5000. The overlap of 1 means the 
system retrieve 1-st stored pattern perfectly. The storage capacity a c is the 
loading rate at Tr m — » in the solid line. 

Theoretical results obtained by SCSNA and computer simulation results 
agree well because the overlap given by computer simulation drastically de- 
creases at a c . 

We can see that the storage capacity in the case without synaptic depression 
is 0.060 in Fig. [5] (a), while in the case with synaptic depression it is 0.048 in 
Fig. [5] (b). Therefore, the synaptic depression decreases the storage capacity 
for finite temperature. 

Next, Fig. [3] shows the phase diagram with respect to temperature T. In 
this phase diagram, the region in which the network succeeds in retrieving 1— st 
memory pattern which is the initial state is denoted as the "Retrieval" phase, 
i.e., ir} n ~ 1. In the "Non- Retrieval" phase, the retrieval is not successful i.e., 
■K m ~ 0. The solid and dashed lines respectively indicate the storage capacity 
a c at corresponding temperature by solving eqs. (|T4| - (fT9|) and (f2Tj) in the cases 
without and with synaptic depression i.e., 7 = 0.0, 7 = 0.5. On the basis of 
this result, we can see that the synaptic depression does not change the storage 
capacity in the low temperature limit, while the storage capacity decreases for 
finite temperature. These results expand the previous result which argue that 
synaptic depression docs not change the storage capacity in the low temperature 
limit [S]. 

Fig. [J] also shows the diagram with respect to the level of synaptic depres- 
sion at a controlled temperature. The solid and the dashed lines indicate the 
cases of T = 0.05 and T = 0.1, respectively. These results indicate that the 
storage capacity decreases when the level of synaptic depression increases in fi- 
nite temperature. Furthermore, we can see that the higher the level of synaptic 
depression, the more storage capacity decreases. 

4 Conclusion 

In this paper, we discussed the associative memory model with synaptic depres- 
sion and analytically investigated the storage capacity at finite temperature by 
SCSNA. There have been previous investigations of how the synaptic depression 
affects the storage capacity[5][S]. However, their investigations focused on only 
low temperature limit. The analysis introduced in this paper enables us to dis- 
cuss the storage capacity of the model with synaptic depression more generally. 
As a result, we found that the synaptic depression decreased the storage capac- 
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ity in the case of finite temperature in contrast to the result in the case of zero 
temperature, where synaptic depression does not change the storage capacity. 



5 Appendix:Detail SCSNA Description 

The goal of this appendix is to derive the order parameter equations from (| 14[) 
to (fTT)| by using SCSNA. By using eq. (fT3"]) . we rewrite the internal potential 
PI) as 

aiV 

The output of ri can be formally expressed by 



= G 



(5^ §**-"') (27) 
2,iW ff< + 2(rb^ ff< )' (29) 



where G(x) represents the solution of x = G(x)D Here, we discuss when the 1-st 
pattern is retrieved, and then the residual overlap ttI/ ~ 0(1/ y/N) (fi + 1 1) 
because a ~ 0(1). Thus, the overlap tt^ can be denoted as 

JV 

E^(-r+^y^r) (3D 

N , N 
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J2$rf+Uir?, (33) 



where 



1 N 



3=1 
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by using the following approximation of eq. (|29[) . 



n ~ ^ + ^^f<rf>, (35) 



= g'\-— — ver<|, (37) 



where rj is the value drawn the effect of /i-th pattern from n. Solving eq. 



(|33|) . the overlap 7r^ can be rewritten as 



JV 



2(1 + 7) V^OO (3 g) 



and the internal potential (|26| is 

ei 1 i w aJV 

2(l + 7 ) + jV(l-L/)|^^ 

el l Af qW 

2(1+7) iV(l-C/)-^f4^ iJ ' 1-C/ 



1^1 



£■71" 

Zi + Tn, (39) 



2(1 + 7) 

N aN 

T = -^-r,. (41) 
1 -u y ' 

where we use the approximation, r\ ~ r,, becasue — fj ~ 0(1/ s/N). The 
first term in the right-hand side of eq. ([3TJ1) comes from signal components, the 
second term is assigned to Gaussian distribution with mean and the following 
variance, 

a 2 = E[( Zl f] - (E[ Zi }) 2 = J^jjyd, (42) 

and the third term means the systematic bias of the crosstalk noises. Here, we 
define Edwards- Anderson order parameter as 



h^r- (43) 



N 
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We obtain the following order parameter equations: 



Y 




+ az + TY) (z~^(0,l)), 



(44) 



(45) 



/ Dz{{Y 2 )), 



(46) 



U 



\ J Dzz((Y)), 



(47) 



r 



aU 



(48) 



1-17' 



aq 



(49) 
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where we use the correspondence, £f — > £, 7r^ — > 7r r , rj — > Y, 71"^ — > 7r m , 
Dz = l/\/27rcxp(— z 2 /2), and the bracket ((•)) represents the average with 
respect to stochastic variable £^ . The above equations return to the usual order 
parameter equations in the case without synaptic depression. 
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Figure 2: Dependency of 7r m on a at T = 0.1. Solid line indicates theoretical 
results by solving (fT^|) - ([TT)l) and ([21]) numerically. Error bars indicate medians 
with minimum and maximum values obtained by computer simulation in 11 
trials at N = 5000. (a): case without synaptic depression, i.e., 7 = 0.0. (b): 
case with synaptic depression, i.e., 7 = 0.5, r = 2.0, U SE — 0.25. 
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Figure 3: Phase diagram representing temperature dependence of storage ca- 
pacity. In retrieval phase, network can retrieve memory pattern, 7r m ~ 1, while 
in non-retrieval phase, network cannot retrieve any memory pattern, n m ~ 0. 
Solid line indicates a c in case without synaptic depression, i.e., 7 = 0.0 Dashed 
line indicates a c in case with synaptic depression, i.e., 7 = 0.5. 




Figure 4: Phase diagram representing level of synaptic depression dependence 
of storage capacity. Solid line indicates a c in case of T = 0.05. Dashed line 
indicates a c in the case of T = 0.1. 
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